Adaptive sensor sampling for power efficient context aware inferences

ABSTRACT

Disclosed is a system, apparatus, computer readable storage medium, and method to perform a context inference for a mobile device. In one embodiment, a data processing system includes a processor and a storage device configurable to store instructions to perform a context inference for the data processing system. Data may be received from at least a first sensor, and a first classification of the data from the sensor may be performed. Confidence for the first classification can be determined and a second sensor can be activated based on a determination that the confidence fails to meet a confidence threshold. A data sample classification from the activated second sensor may be classified jointly with the data from first sensor.

FIELD

The subject matter disclosed herein relates generally to powermanagement techniques and sensor activation scheduling.

BACKGROUND

Electronic devices can be equipped with a variety of sensors and inputsto monitor and discover information about a device's environment. Forexample, a device may utilize an accelerometer to measure aspects ofdevice movement or a microphone to measure the device's nearby audioenvironment.

Programs or applications running on a device may process data fromsensors in order to provide an enhanced user experience. A device caninfer context based on the sensor data. However, sensor data maysometimes be ambiguous or misleading, and false positives can occur wheninferring context based on output from a single sensor. However,activating multiple data sensors can be detrimental to a goal of powerefficiency.

Therefore, new and improved sensor management techniques are desirable.

SUMMARY

Embodiments disclosed herein may relate to a method for performing acontext inference for a mobile device. The method includes receivingdata from at least a first sensor, performing a first classification ofthe data from the at least first sensor, and determining a confidencevalue for the first classification. The method further includesactivating a second sensor based on a determination the confidence valueof the first classification fails to meet a confidence threshold andperforming a second classification of a second data sample from thesecond sensor jointly with the data from the at least first sensor.

Embodiments disclosed herein may also relate to a machine readablenon-transitory storage medium with instructions to perform a contextinference for a mobile device. The method includes receiving data fromat least a first sensor, performing a first classification of the datafrom the at least first sensor, and determining a confidence value forthe first classification. The method further includes activating asecond sensor based on a determination the confidence value of the firstclassification fails to meet a confidence threshold and performing asecond classification of a second data sample from the second sensorjointly with the data from the at least first sensor.

Embodiments disclosed herein may also relate to an apparatus thatincludes means for receiving data from at least a first sensor, meansfor performing a first classification of the data from the at leastfirst sensor, and means for determining a confidence value for the firstclassification. The apparatus further includes means for activating asecond sensor based on a determination the confidence value of the firstclassification fails to meet a confidence threshold and means forperforming a second classification of a second data sample from thesecond sensor jointly with the data from the at least first sensor.

Embodiments disclosed herein may further relate to a data processingsystem including a processor and a storage device configurable to storeinstructions to perform a context inference for the data processingsystem. The instructions cause the processor to receive data from atleast a first sensor, perform a first classification of the data fromthe at least first sensor, and determine a confidence value for thefirst classification. The instructions further cause the processor toactivate a second sensor based on a determination the confidence valueof the first classification fails to meet a confidence threshold andperform a second classification of a second data sample from the secondsensor jointly with the data from the at least first sensor.

Other features and advantages will be apparent from the accompanyingdrawings and from the detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system in which aspects of the inventionmay be practiced;

FIG. 2 illustrates a flow diagram of a context aware sensor manager, inone embodiment;

FIG. 3 illustrates a chart of sensor data classifications, in oneembodiment;

FIG. 4 illustrates a flow diagram of sensor activation upon detection ofspecified classification transitions, in one embodiment; and

FIG. 5 illustrates a flow diagram of a context aware sensor manager, inanother embodiment.

DETAILED DESCRIPTION Device Overview

FIG. 1 is block diagram illustrating an exemplary data processing systemin which embodiments of the invention may be practiced. The system maybe a device 100, which may include one or more processors 101, a memory105, I/O controller 125, and network interface 110. The device 100 mayalso include a number of device sensors coupled to one or more buses orsignal lines further coupled to the processor 101. It should beappreciated that the device 100 may also include a display 120, a userinterface (e.g., keyboard, touch-screen, or similar devices), a powerdevice 121 (e.g., a battery), as well as other components typicallyassociated with electronic devices.

In some embodiments the device 100 may be a mobile or non-mobile device.The network interface 110 may also be coupled to a number of wirelesssubsystems 115 (e.g., Bluetooth, WiFi, Cellular, or other networks) totransmit and receive data streams through a wireless link to/from awireless network, or may be a wired interface for direct connection tonetworks (e.g., the Internet, Ethernet, or other wireless systems).Thus, the device 100 may be a: mobile device, wireless device, cellphone, personal digital assistant, mobile computer, tablet, personalcomputer, laptop computer, or any type of device that has processingcapabilities.

The device 100 can include sensors such as a proximity sensor 130,ambient light sensor (ALS) 135, accelerometer 140, gyroscope 145,magnetometer 150, barometric pressure sensor 155, and/or GlobalPositioning Sensor (GPS) 160. In some embodiments, microphone 165,camera 170, and/or the wireless subsystem 115 are used as sensors toanalyze the environment of the device 100. For example, the microphone165 can provide audio data for determining whether the device 100 isinside a moving automobile or in a quiet office. Each sensor may beimplemented on the same or different subsystems. When implemented ondifferent and separate subsystems, each sensor may have dedicated powersuch that each sensor can be independently powered or unpowered.Independent separate subsystem sensors may still communicate with eachother and other components in the device 100 over the signal lines orbuses described above.

The memory 105 may be coupled to the processor 101 to store instructionsfor execution by processor 101. In some embodiments, the memory 105 isnon-transitory. The memory 105 may also store one or more models ormodules to implement embodiments described below. The memory 105 mayalso store data from integrated or external sensors. In addition, thememory 105 may store application program interfaces (APIs) for accessingone or more modules 171 (e.g., a Content Aware Sensor Manager module)described in greater detail below. The memory 105 may also storeconfiguration files describing pre-defined feature schedulinginformation, training data sample, sensor data sample, or power profilesrelated to classes or features.

It should be appreciated that embodiments of the invention as will behereinafter described may be implemented through the execution ofinstructions, for example as stored in the memory 105 or other element,by processor 101 of device 100 and/or other circuitry of device 100and/or other devices. Particularly, circuitry of device 100, includingbut not limited to processor 101, may operate under the control of aprogram, routine, or the execution of instructions to execute methods orprocesses in accordance with embodiments of the invention. For example,such a program may be implemented in firmware or software (e.g. storedin memory 105 and/or other locations) and may be implemented byprocessors, such as processor 101, and/or other circuitry of device 100.Further, it should be appreciated that the terms processor,microprocessor, circuitry, controller, etc., may refer to any type oflogic or circuitry capable of executing logic, commands, instructions,software, firmware, functionality and the like.

Further, it should be appreciated that some or all of the functions,engines or modules described herein may be performed by device 100itself and/or some or all of the functions, engines or modules describedherein may be performed by another system connected through I/Ocontroller 125 or network interface 110 (wirelessly or wired) to device100. Thus, some and/or all of the functions may be performed by anothersystem and the results or intermediate calculations may be transferredback to device 100. In some embodiments, such other device may comprisea server configured to process information in real time or near realtime. In some embodiments, the other device is configured topredetermine the results, for example based on a known configuration ofthe device 100.

Content Aware Sensor Manager Overview

Device sensors can provide contextual data about a user and environment.Context, as used herein may be any information that can be used tocharacterize the situation of a user. For example, context can describewhat a user is doing, the user's environment/surroundings, where theuser is, or what intentions the user may have.

In one embodiment, a Content Aware Sensor Manager (CASM) (e.g.,implemented as a module 171 or engine can opportunistically turn on,power up, or activate additional sensors on the same or othersubsystems) to improve performance of context aware inferences. In someinstances, a single sensor data sample may be insufficient to provide anaccurate context inference or classification. Therefore, one or moreadditional sensor streams may be activated.

For example, based on an accelerometer data sample, the CASM or adedicated classifier may determine the mobile device is placed on thesurface of a desk or table. The accelerometer may determine orientationof the surface of the device with respect to gravity. Orientation may becharacterized by angle measurements of pitch and roll. Based on theorientation of the device (e.g., angled consistent with a flat on a deskplacement) and/or a lack of detectable motion recorded by theaccelerometer, the CASM or dedicated classifier may infer the device isstationary on a desk surface. However, the confidence in the placementon the desk classification may be low, and the CASM can activateadditional sensors to verify the classification. For example, activatingand classifying data from the GPS 160 may infer that the user is on apublic road with the mobile device placed on a car dashboard. When theuser is on a street or road, the user and mobile device are unlikely tobe at a desk. The combination of the accelerometer 140 and the GPS 160can have an associated joint confidence value indicating whether theuser is most likely (i.e., has a high confidence) to be at a desk ordriving in a car. In some cases, the additional sensor data andclassification for two or more sensors can provide a higher confidencein a particular classification.

In another example, the accelerometer may determine a user has themobile device in a pocket, as indicated by the angle consistent withpocket placement. Again, the CASM, may not have high enough confidenceto determine whether the user is sitting in a work environment (e.g., ata desk), or sitting in a car while the car is momentarily at rest. Asabove, additional sensors may be activated and classified in order toresolve classification ambiguity. Further details are described ingreater detail below.

The CASM or a dedicated classifier can classify one or more sensorstreams (e.g., in one or more classification iterations) until a finaldetermination of context for the user is determined. In one embodiment,the CASM can process or compute features from a sensor data sample, andoutput a classification (e.g., state or context aware inference). Forexample, as CASM receives sensor data, meaningful features representingthe discriminatory aspects of the data can be computed and matched to amodel to determine a best-fit classification. Because a sensor datasample may provide ambiguous or inaccurate classifications, the CASM maytrigger or activate one or more additional sensors to verify orstrengthen an initial classification, or when a confidence associatedwith the classification is below a pre-determined threshold.

In one embodiment, Bayesian probabilities can be used in modelclassifications. For example, the CASM may determine which class has thehighest probability and output the maximum probability as theconfidence. Alternatively, the CASM may output the difference betweenthe maximum probability classification and the next most probable (i.e.,second maximum) classification as the probability. For example, a firstclassification probability may be 0.6 and a second classificationprobability may be 0.4, where 1 represents a certainty of occurrence,and 0 represents a certainty of non-occurrence. In this previousexample, the difference between the first classification probability of0.6 and the second classification probability of 0.4 would be 0.2, arelatively low confidence. In a second example, the probability for afirst classification is 0.9 and the probability of a secondclassification is 0.1. In this second example, the difference betweenthe two classifications is 0.8, which may be a relatively highconfidence.

In other embodiments, support vector machines may be used to determinedata points. For example, a nearest neighbor classifier may computefeature vectors and find the nearest feature vector from a knownuniverse of possible data points. By referencing a historical or knownset of data points, a particular classification may be assigned. Forexample, a feature vector may match a nearest point and the point can beassigned or associated with a specific classification. When a featurevector cannot match a nearest point or if the feature vector cannot beassociated with a specific classification, additional sensors may betriggered or activated. The additional sensor data may be classifiedbased on a nearest neighbor classifier as described above.Alternatively, more sensors may be triggered/activated until aclassification match can be determined.

FIG. 2 illustrates a flow diagram of the CASM, in one embodiment. Atblock 205, the CASM can receive data from at least a first sensor (e.g.,a proximity sensor 130, ALS 135, accelerometer 140, gyroscope 145,magnetometer 150, barometric pressure sensor 155, GPS 160, microphone165, and camera 170).

At block 210, the CASM or a classifier can perform a classification ofthe data from at least the first sensor. For example, the first datasensor may output a first data sample for processing. The CASM can run afeature computation on the first data sample and the feature computationmay be compared to a predetermined training data set to output aclassification. In one embodiment, the first data sample provides atemporary classification and the CASM may determine to provide thetemporary classification as the final output classification based on aconfidence in the temporary classification. Further details of featurecalculation are described below.

At block 215, the CASM can determine a confidence value for the firstclassification. The confidence value may be based on one or more factorsas described in greater detail below.

At block 220, the CASM can activate a second sensor based on adetermination the confidence value of the first classification fails tomeet a confidence threshold. For example, given a low confidence in aprevious classification, the CASM may activate an additional sensor andrun another classification determination using an additional sensordifferent from the sensor used in the previous classification.

At block 225, the CASM can perform a second classification of a seconddata sample from the activated second sensor jointly with the data fromat least the first sensor. In one embodiment, the classification of theadditional sensor data also has an associated individual confidence, oran overall confidence of both the first and second data classifications.In one embodiment, the first and second sensor data classifications maybe jointly classified in order to determine a final/outputclassification and associated confidence value. In some embodiments, twosensor data samples may be classified separately and independently, twothe different classifications are provided and the CASM may output thehighest confidence value of the two independent classifications (fromthe first and second sensors). In other embodiments, the CASM mayautomatically output the second sensor classification based on adetermination that classifying the first classification did not meet anassociated confidence threshold. In yet other embodiments, the CASM mayoutput all determined classifications and their associated confidenceand a program or application may further process the results to make anindependent determination of classification and ultimately infercontext. Classification and confidence generation are described ingreater detail below.

Sensor Data

The CASM may process or compute data received from one or more sensorsattached to the mobile device (e.g., a proximity sensor 130, ALS 135,accelerometer 140, gyroscope 145, magnetometer 150, barometric pressuresensor 155, GPS 160, microphone 165, and camera 170) to reportinformation related to a device's environment. In some embodiments, theCASM can receive external sensor data from communicatively connectedexternal devices (e.g., via a USB connection or WiFi connection to anexternal camera) through the I/O controller 125. The CASM or a dedicatedclassifier (e.g., an always-on or frequently active classifier) canclassify sensor data.

The CASM or a dedicated classifier can receive raw sensor data for usein classification (i.e., feature computation) as described below.Alternatively, an intermediary device or program can pre-process sensordata before feature computation. For ease of description, sensor data asused herein refers to unprocessed data (e.g., data received from anaccelerometer 140, ambient light sensor 135, microphone 165 or othersensor). For example, data from an accelerometer 140 may have attributesof time, acceleration along an x-axis, acceleration along a y-axis, andacceleration along a z-axis. However, a person of ordinary skill in theart will recognize that sensor data may be received and processed inother forms other than the examples given here.

As used herein, a data sample can include a portion of data received oroutput from a sensor (e.g., the sensors of the device 100). A sensordata sample may be sampled at a particular frequency (e.g., 50 Hz, 60 Hzor other rate depending on the sampling device and the datarequirements). Feature computation may be performed on a moment, slice,or window of time selected from a stream or set of sensor data. Forexample, device 100 may compute features over a 1 second time periodselected from a longer stream (e.g., a 10 second time period, constantlyon sensor, or some other time period). For example, raw accelerometer140 data may be sampled at 60 Hz such that 1 second of data provides 603-dimensional accelerometer 140 vector samples in the x-axis, y-axis,and z-axis for a net input size of 180 samples.

In one embodiment, the CASM or a dedicated classifier, can compare asensor data sample to a training data sample to classify the sensor datasample. For example, in a test/training setting, sensors may receive avariety of inputs to cover possible real-world contexts, and theresulting feature computations and classifications can be recorded.Using this pre-computed historical data, a current sensor data samplecan be compared to estimate a most likely classification as well as forassigning the classification a confidence.

In other embodiments, instead of directly using training data toclassify the data (e.g., nearest neighbor classifier), modelspre-computed from the training data may be used (e.g., Support VectorMachines, Neural Network, Decision Tree, Bayesian Classifier, etc.).

In one embodiment, accuracy of classifications may improve if multiplesensors of different types are used. For example, two or more sensorswith a joint classification may nonetheless result in a low confidenceclassification (e.g., microphone and accelerometer may not be enough todetermine a high confidence meeting classification). Therefore,additional sensors (e.g., GPS or other motion sensor) may be triggeredor activated to further boost confidence in a final classification.

In one embodiment, the sensor data samples may be from the same ordifferent sensor subsystems of the device 100. For example, theaccelerometer may be on a separate sensor subsystem to receive powerwhile a GPS on a different subsystem is unpowered.

In one embodiment, the CASM may determine whether a classificationconfidence threshold is met based on the computed feature. In oneembodiment, the threshold confidence can be determined by comparing thefeature with a training data set. Confidence is discussed in greaterdetail below.

In one embodiment, the CASM has access to power profiles of availablesensors. The power profile may estimate the average or likely powerusage requirements of a particular sensor so that the CASM can order thesequence of sensor activations from least power usage to greater powerusage. For example, the accelerometer 140, clock, and ambient lightsensor 135 may have a power profile identifying it as a low power use.Conversely, the WiFi sensor and GPS 160 may have a power profileidentifying them as a high power use. Therefore, the CASM may determinethat the accelerometer 140 should be prioritized for activation over theGPS 160, or an ambient light sensor 135 over WiFi, all else being equal.In another example, the gyroscope 145 and microphone 165 may be inbetween the accelerometer 140 and GPS 160 with regards to power usage.Therefore, in one example, the accelerometer 140 may be set to be thefirst initial sensor that is always on or first to activate, thegyroscope 145 may be the second sensor activated to strengthen aclassification determination, and the third sensor may be the GPS 160when all other sensors are unable to be leveraged for a high confidenceclassification.

In other embodiments, the power profile of a sensor is one of a numberof factors to determine which first or additional sensor is activated.For example, a sensor data sample from a gyroscope 145 may contain lessvaluable context inference data than for example the GPS 160, therefore,the gyroscope 145 may be bypassed such that the GPS 160 is activatedinstead. The value determination may vary based on an initialclassification. For example, the initial classification may be ambiguousas to whether the classification is in a meeting, and additional sensorssuch as a microphone or accelerometer may be more useful than a GPSbecause indoor signal reception may be poor.

Also, the CASM may determine that a variety of sensors of differenttypes are helpful to determine context despite their powercharacteristics. For example, to determine whether a user is in ameeting, the clock, accelerometer 140, and microphone 165 can all beuseful to infer context, however each one in isolation may provide anincomplete picture.

In other embodiments, the CASM may activate two or more sensorsinitially and activate additional sensors at a later point in time. Forexample, the accelerometer 140 and the gyroscope 145 sensors may beactive initially, and the GPS 160 may be the additional sensor activatedwhen a classification ambiguity is detected. In another example, theclock and the accelerometer 140 may be always on in a device, and theWiFi sensor may be activated at a later time to resolve classificationambiguity.

In one embodiment, the CASM may automatically trigger a hardwareinterrupt to power a second sensor on a different subsystem. Forexample, upon determining a low confidence classification, the CASM maytrigger or power up other sensors on different sensor subsystems.

Feature Classification

In one embodiment, the CASM can compute, process, or extract one or morefeatures from sensor data as disclosed above (e.g., raw data from amicrophone 165, accelerometer 140, ambient light sensor 135 or otherinput). Features described herein refer to the result or output ofcomputations executed on sensor data. Features can be used to classify adata set. For example, the CASM can receive as input, raw sensor data(e.g., a sensor data sample) from accelerometer 140, and calculate meanand/or standard deviation. The CASM can use the mean and/or standarddeviation to classify or provide context to the sensor data. Forexample, the classification of a motion sensor may be: walking, sitting,running, or driving, just to name a few. In one embodiment, the featuresare the values, outputs, or results from computer code executed inmemory 105 by a processor 101.

In one embodiment, the CASM may jointly classify two or more sensor datasamples from two or more different sensors. For example, the CASM mayread data from the accelerometer and the GPS and provide aclassification (e.g., running, driving in a car, or other motion basedclassification) using the two different sensor data samples in a jointclassification. The two different sensor data samples may be sampledfrom the same window or range of time. In another example, the CASM mayfirst classify data from a first sensor, and determine theclassification does not meet a confidence threshold. Upon activating asecond sensor, the CASM can classify the first sensor data jointly withthe activated second sensor to determine a final classification based onsensor data received from both sensors.

In some embodiments, the CASM may call a separate classifier (e.g., aseparate classifier module from the CASM) to compute features. Aclassifier can take features and compare them to a model to produce aninference (e.g., to determine a most likely classification or state).The separate classifier may be able to process multiple sensor streams.Alternatively, there may be a separate classifier for each type ofsensor. The CASM as described herein may include an integratedclassifier such that the CASM can produce a classification or stateoutput or the classifier may be separate and independent.

Confidence

As described above, the CASM can associate a confidence (e.g.,confidence value or level) with each individual data sensor sampleclassification or joint data sensor sample classification. The CASM mayuse confidence as a way to determine whether additional sensors shouldbe activated. The confidence may be compared to a confidence thresholdand classifications not meeting or exceeding the confidence thresholdcan be verified against a second classification from an additionalsensor data sample.

In some embodiments, in addition to the individual confidence values forthe first and second classifications described above, an overallconfidence may also be associated with the combination of two or moresensor classifications. In some embodiments, the CASM may maintain acumulative overall confidence taking into account multiple individualconfidence values from one or more sensor data classifications. Thecumulative overall confidence may be initiated upon a second sensoractivation, upon a first confidence failure, or upon determination of asecond classification confidence. For example, upon determining a firstand second data sensor classification and associated confidences, thesecond confidence may be a confidence in the second data sensorclassification only, and an overall confidence (e.g., a third confidencein this example) represents the confidence of the first and secondclassifications together. Because joint classifications classify two ormore sensor data samples, confidence of a joint classification mayresult in one confidence value similar to an overall confidence asdescribed above (e.g., instead of an independent confidence in additionto an overall confidence).

The CASM may use the overall confidence to determine whether to continueto activate additional sensors. For example, a first sensor data samplemay result in a low confidence classification, and a second sensor datasample may also result in a low confidence classification. However, theCASM may in some embodiments, determine that when two or more sensordata samples infer the same classification the overall confidence maystill be high enough to pass the confidence threshold.

In another example, a first sensor data sample may result in a lowconfidence classification, and a second sensor data sample may result ina high confidence classification. In this example, the CASM maydetermine that second sensor data sample is more likely to provide acorrect classification based on the failure of the first data sample toovercome a minimum confidence. Therefore, the CASM may ultimately outputa final determination classification based on the second sensor datasample.

In one embodiment, the confidence threshold is determined based on apredetermined value based on historical or training data. For example,training data may indicate a threshold confidence that provides an idealbalance between power consumption and context inference performance. Insome embodiments, a user may adjust the threshold to achieve a differentbalance. In yet other embodiments, the CASM may independently adjust thethreshold based on what type of sensor is used, the number of sensors,or other factors.

Confidence can be computed based on the distance of the current datapoint (e.g., a data point output from a feature calculation) to thenearest classification, with small distances corresponding to highconfidence and large distances corresponding to low confidence.

The measure of distance may vary. For example, a GPS 160 clusteringalgorithm that attempts to associate the user's current location with anearest cluster can use the geographic distance from the user to thecluster centroid as a measure of distance.

For a classifier that outputs probabilities or likelihoods, a confidencecan be computed based on these output probabilities or likelihoods. Forexample, for a classifier that outputs posterior probabilities P_(i) foreach class, a confidence can be computed based on the maximumprobability, e.g.

$C = {\max\limits_{l = {({1,2,\mspace{11mu}\ldots}\mspace{14mu})}}P_{i}}$

For a classifier that outputs likelihoods L_(i) for each class, aconfidence can be computed based on the normalized relative likelihoods.For example,

$C = \frac{{\log\; L_{i\; 1}} - {\log\; L_{i\; 2}}}{{{\log\; L_{i\; 1}}} + {{\log\; L_{i\; 2}}}}$where i1 is the index of the most likely class, and i2 is the index ofthe second most likely class.

Ambiguous Classifications

In one embodiment, the CASM determines a classification or state basedon the sensor sample data. In other embodiments, the classifier outputsa state or classification to the CASM. For example, if the classifieroutputs classification A which is known to be typically confused withstate B, the CASM can activate or enable an additional sensor stream toverify the classification is correct.

In one embodiment, the CASM determines which additional sensor toactivate based on the ambiguous classifications. For example, confusionbetween classification A and B may result in activating on sensor C toresolve the specific ambiguity. In another example, confusion betweenclassification A and D may result in activating on sensor E to resolvethis ambiguity.

FIG. 3 illustrates a chart of sensor data classifications, in oneembodiment. FIG. 3 illustrates three classification clusters (e.g., asbased on prior calculated training data samples). Class A 315, Class B325, and Class C 335 are three possible classifications based on featurecalculation from one sensor (e.g., a first activated sensor). The X 340and Y 305 axes may represent the resulting features from featurecalculations performed on the raw data from a sensor. Clusters ofclassifications based on historical or prior training data can be usedby the CASM to determine the confidence of a current/new classification.For example, if a point 310 (e.g., point representation of aclassification from a sensor data sample) lies within a known range orcluster of Class A historical classifications, then the CASM may have ahigh confidence that the point 310 is Class A and output the contextinference associated with Class A without activating additional sensors.

If a point (e.g., the point 320) lies outside of any cluster but isclosest to a particular cluster (e.g., point 320 closest to Class C 315despite being outside of the cluster), the CASM may output a lowconfidence classification (e.g., a temporary classification) which wouldstill trigger a additional/second sensor. Upon triggering or activatingthe second sensor, the CASM or classifier may classify the data based oneither the second sensor data alone or jointly classify the first andsecond sensor data together. If the joint or second sensorclassification results in a higher confidence classification (e.g.,results in features inside Class A 315, Class B 325, or Class C 335clusters), the higher confidence classification may be chosen for outputby the CASM. If a temporary classification was generated, the temporaryclassification may be overridden by the most recent joint or independentclassification based on the additional sensor data sample. If theadditional sensor data sample agrees with the nearest classificationestimation, the agreed classification is outputted with higherconfidence than if only one sensor or data sample classified the data.If the additional sensor data sample results in a differentclassification than the first data sample, the CASM may activateadditional sensors in addition to the second sensor, or alternativelymay make a determination as to which of the classifications to outputfor example based on the relative confidence levels of each individualclassification.

If a point (e.g., point 330) lies outside of a known cluster (e.g.,Class A 315, Class B 325, or Class C 335), then the CASM may have a lowconfidence that the point can be accurately classified. In someembodiments, if the point is equidistant from two or more nearestclassifications (e.g., Class A 315, Class B 325, or Class C 335 are allequidistant for the point 330 in this example), the CASM may not outputany classification and trigger an additional/second sensor. Upontriggering/activating the second sensor, a joint or independent datasensor classification can be obtained as described above. For example,if the joint or independent classification is high confidence (e.g.,within a cluster), the classification is chosen and output by the CASM.Otherwise, if there is still some ambiguity as to the classification(e.g., low confidence), then additional sensors may be activated andfurther classifications may be determined

Ambiguous Change of Classification

In one embodiment, the CASM can make a determination to turn onadditional sensors based on a change in classification. Thedetermination to turn on additional sensors based on a change inclassification may be independent from the confidence of the particularclassification.

The CASM may determine that a user is unlikely to transition directlyto/from certain classifications. For example, the CASM may process adata sensor sample from an accelerometer 140 that infers a user may bedriving in a car based on sensor data sampled at time window T₁. If atthe next classification from the same sensor but at the next time windowT₂, the classification infers the user is sitting at a desk, the CASMcan determine that this direct transition (i.e., car to desk) isunlikely. Upon determining the transition is unlikely the CASM maydetermine the overall classification confidence to be low, or may lowerthe individual confidence associated with the most recentclassification. Users are more likely to have an intermediate transitionbetween driving to an office, such as getting out of the car, standing,and then walking from the car to the office before resuming a sittingposition at a desk. Therefore, the CASM may activate an additionalsensor, such as a microphone 165, and classify the sensor data sample todetermine if the user is still in the car. If the microphone 165 sensordata sample is unable to provide a high confidence classification, theGPS 160 may be activated. After a verification classification can bedetermined (e.g., with one or more additional sensor activations afterthe improbable transition is detected) the CASM can make an ultimatedetermination of classification (e.g., the user has indeed left the carand is sitting at a desk, or the user is still in the car). Therefore,the CASM can activate additional sensors before providing an erroneousor inaccurate final output classification.

In one embodiment, the CASM has access to (e.g., in device 100 memory)probable and improbable classification transitions (e.g., a statediagram, database or flat file of transitions). For example,transitioning from driving to/from walking and sitting at a desk to/fromwalking are just two example probable transitions. However, drivingto/from sitting at a desk is an improbable transition when no otherintermediate transitions or classifications have been detected.

In another example, a user may be in a meeting as determined byclassifying a data sample from an audio source (e.g., microphone 165)when no further audio is present in the audio source. The CASM canactivate an accelerometer 140, GPS 160 or other motion based sensor toverify the user exited or is in the process of exiting the meeting.Otherwise, if the user and device 100 are stationary, the confidencethat the user is still in the meeting may be increased or maintained.

In one embodiment, the internal clock of the device 100 is used as anadditional sensor. The CASM may also use the internal clock as a timesensor to determine how long a user is in a meeting. If a recordedduration of time is too short, too long, or the meeting ended after atime period not likely to be a typical meeting length, the confidence ofthe classification “in meeting” may be lowered. For example, typicalmeetings may last for at least 15 minutes, and may often last 30 minutesto one hour. Meetings less than 15, meetings more than two hours, ormeetings ending after a duration other than 15 minute increments maycause the CASM to activate additional sensors and compare theclassification of the additional sensor(s) to the first sensorclassification.

FIG. 4 illustrates a flow diagram of sensor activation upon detection ofspecified classification transitions, in one embodiment. In oneembodiment, an initial active sensor may output sensor data samples tothe CASM or dedicated classifier. The sensor data samples may be sampledover a period of time (e.g., sample time periods of: t₀ 412 to t₁ 422,t₁ 422 to t₂ 432, t₂ 432 to t₃ 442, t₃ 442 to t₄ 452, t₅ 452 to t₅ 462,and t₅ 462 to t₆ 472).

During the first time window t₀ 412 to t₁ 422, Sensor X 405 hasclassified the data sample as Class A 415, and Sensor Z 410 is inactive420 (e.g., no power is applied to the sensor, therefore no Sensor Z 410data sample is available during time window t₀ 412 to t₁ 422).

During the next time window, t₁ 422 to t₂ 432, Sensor X 405 outputs asensor data sample indicating a Class B 425 classification. Sensor Z 410is still inactive, as the CASM has just finished processing the datasample from window t₁ 422 to t₂ 432. In one embodiment, the CASM cancompare a current classification to an immediately prior in time (t₀ 412to t₁ 422) classification to detect unlikely or improbableclassification changes or transitions. In this example, the CASMdetermines that the output from Sensor X 405 had an immediately prior intime (t₀ 412 to t₁ 422) classification of Class A and a currentclassification of Class B. In this example, the transition from Class Ato Class B was predetermined as unlikely or improbable (e.g., Class Amay be driving in a car, and Class B may be sitting at a desk). Upondetecting an unlikely or improbable change in classification hasresulted from a sensor (e.g., Sensor X 405), the CASM can activate anadditional sensor (e.g., Sensor Z 410) to verify the currentclassification. In this current example, the CASM activates Sensor Z 410after the unlikely or improbable class transition (Class A to Class B)is detected (e.g., at time t₂).

During the next time window t₂ 432 to t₃ 442, Sensor X 405 outputs asensor data sample resulting in a Class B 425 classification. The CASMclassifies the sensor data sample from Sensor Z 410 as Class A 435during window t₂ 432 to t₃ 442. In one embodiment, the CASM suppressesthe Class B 435 determination from Sensor X 405 to output a Class A 440output. In one embodiment, the CASM may ignore the Sensor X 405classifications until a new transition is detected.

During the next time window t₃ 442 to t₄ 452, Sensor X 405 data isclassified as Class A 445 and the CASM may de-activate Sensor Z 410(e.g., after time t₄). In this case, the CASM may determine thetransition from Class B to Class A triggers the activation of Sensor Z410 again. However, the CASM may grant exceptions to the general rule ofdetecting an improbable transition when returning from a priortransition that was also improbable. In the current example, Class A 415to Class B 425 was an improbable change in classification causing theCASM to trigger an additional sensor to verify the class change.However, when the initial sensor reverted back to Class A 445 from ClassB 435, the CASM may not trigger the activation of Sensor Z 410 becausethe Class B 435 classification is treated as an erroneous readingignored by the CASM. In some embodiments, the CASM triggers anadditional sensor when a high confidence classification transitions to alow confidence classification, but may not trigger additional sensorswhen a low confidence classification transitions to a high confidenceclassification.

In some embodiments, one or more additional sensors may remain activefor a pre-determined or user determinable number of window samplesbefore de-activating. For example, a configuration setting for powerconservation may immediately to turn off additional sensors after theyprovide a verification classification to the CASM. Conversely, aconfiguration setting prioritizing performance over energy conservationmay cause additional sensors to remain active for a number of additionalcycles or windows beyond the end of an ambiguous classification orunlikely transition.

During the next time window t₄ 452 to t₅ 462, the sensor data samplefrom Sensor X 405 is classified as Class C 455, which is a possibleclassification transition that does not trigger an additional sensor.

During the next time window t₅ 462 to t₆ 472, the sensor data samplefrom Sensor X 405 is classified as Class C 465, which is aclassification transition from Class A 414 that does not trigger anadditional sensor. Therefore, Sensor Z 410 remains inactive because itwas not triggered at time t₅.

Content Aware Sensor Manager

The CASM may be initiated based on the feature vector computed from theraw sensor data. For example, if the computed feature vector lies in anambiguous region of the feature space, additional sensor streams can beenabled. The distance of the feature vector from the nearest classes canbe used for this determination. For example, the standard deviation ofthe accelerometer 140 norm and the average orientation of the device mayindicate the data can be generated either by walking or driving, withthe distance of the feature vector to the centroid of each of theseclasses being small and similar.

The CASM may also be initiated based on performing a confidence test onclassifier outputs and enabling additional sensor streams (residing onthe same or other subsystems) if the reported confidence is below apre-determined threshold.

FIG. 5 illustrates a flow diagram of the CASM, in one embodiment. Atblock 505, the CASM can receive or access sensor data (e.g., a sensordata sample).

At block 510, the CASM can attempt to classify the sensor data receivedin the previous block 505. Data classification may involve featurecalculation as described above.

At block 515, the CASM can determine a confidence associated with thedetermined classification from the previous block 510. The confidencemay be based on one or more of the factors described above.

At block 520, the CASM can determine whether the confidence associatedwith the sensor data meets or exceeds a confidence threshold. Forexample, the confidence threshold may be a pre-determined confidencelevel or value known to provide for accurate classifications. Lowconfidence may indicate the classification determined in block 510 isinaccurate or untrustworthy. If the CASM determines the classificationdoes meet or exceed a minimum level of confidence, a final determinationis output at block 530. Alternatively, if the CASM determines theclassification does not meet a pre-defined minimum level of confidence,the CASM can activate one or more additional sensors at block 525 andthe CASM can classify the additional sensor data at block 510.

At block 530, the CASM provides a classification based on the one ormore sensor data samples classified in the previous blocks. In oneembodiment, the CASM determines that a classification determined by asubsequent sensor data sample overrides any earlier initialclassification determination.

The output of the CASM may be used by an application (e.g., a programrunning on the device 100, such as an exercise program, activity logger,calendar application, or other software to read processed sensor dataand associated classifications). The CASM may also output an estimatedconfidence associated with each classification.

The word “exemplary” or “example” is used herein to mean “serving as anexample, instance, or illustration.” Any aspect or embodiment describedherein as “exemplary” or as an “example” is not necessarily to beconstrued as preferred or advantageous over other aspects orembodiments.

It should be appreciated that when the device 100 is a mobile orwireless device that it may communicate via one or more wirelesscommunication links through a wireless network that are based on orotherwise support any suitable wireless communication technology. Forexample, in some aspects computing device or server may associate with anetwork including a wireless network. In some aspects the network maycomprise a body area network or a personal area network (e.g., anultra-wideband network). In some aspects the network may comprise alocal area network or a wide area network. A wireless device may supportor otherwise use one or more of a variety of wireless communicationtechnologies, protocols, or standards such as, for example, CDMA, TDMA,OFDM, OFDMA, WiMAX, and Wi-Fi. Similarly, a wireless device may supportor otherwise use one or more of a variety of corresponding modulation ormultiplexing schemes. A mobile wireless device may wirelesslycommunicate with other mobile devices, cell phones, other wired andwireless computers, Internet web-sites, etc.

The teachings herein may be incorporated into (e.g., implemented withinor performed by) a variety of apparatuses (e.g., devices). For example,one or more aspects taught herein may be incorporated into a phone(e.g., a cellular phone), a personal data assistant (PDA), a tablet, amobile computer, a laptop computer, a tablet, an entertainment device(e.g., a music or video device), a headset (e.g., headphones, anearpiece, etc.), a medical device (e.g., a biometric sensor, a heartrate monitor, a pedometer, an Electrocardiography (EKG) device, etc.), auser I/O device, a computer, a server, a point-of-sale device, anentertainment device, a set-top box, or any other suitable device. Thesedevices may have different power and data requirements and may result indifferent power profiles generated for each feature or set of features.

In some aspects a wireless device may comprise an access device (e.g., aWi-Fi access point) for a communication system. Such an access devicemay provide, for example, connectivity to another network (e.g., a widearea network such as the Internet or a cellular network) via a wired orwireless communication link. Accordingly, the access device may enableanother device (e.g., a Wi-Fi station) to access the other network orsome other functionality. In addition, it should be appreciated that oneor both of the devices may be portable or, in some cases, relativelynon-portable.

Those of skill in the art would understand that information and signalsmay be represented using any of a variety of different technologies andtechniques. For example, data, instructions, commands, information,signals, bits, symbols, and chips that may be referenced throughout theabove description may be represented by voltages, currents,electromagnetic waves, magnetic fields or particles, optical fields orparticles, or any combination thereof.

Those of skill would further appreciate that the various illustrativelogical blocks, modules, circuits, and algorithm steps described inconnection with the embodiments disclosed herein may be implemented aselectronic hardware, computer software, or combinations of both. Toclearly illustrate this interchangeability of hardware and software,various illustrative components, blocks, modules, circuits, and stepshave been described above generally in terms of their functionality.Whether such functionality is implemented as hardware or softwaredepends upon the particular application and design constraints imposedon the overall system. Skilled artisans may implement the describedfunctionality in varying ways for each particular application, but suchimplementation decisions should not be interpreted as causing adeparture from the scope of the present invention.

The various illustrative logical blocks, modules, and circuits describedin connection with the embodiments disclosed herein may be implementedor performed with a general purpose processor, a digital signalprocessor (DSP), an application specific integrated circuit (ASIC), afield programmable gate array (FPGA) or other programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination thereof designed to perform the functions described herein.A general purpose processor may be a microprocessor, but in thealternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

The steps of a method or algorithm described in connection with theembodiments disclosed herein may be embodied directly in hardware, in asoftware module executed by a processor, or in a combination of the two.A software module may reside in RAM memory, flash memory, ROM memory,EPROM memory, EEPROM memory, registers, hard disk, a removable disk, aCD-ROM, or any other form of storage medium known in the art. Anexemplary storage medium is coupled to the processor such the processorcan read information from, and write information to, the storage medium.In the alternative, the storage medium may be integral to the processor.The processor and the storage medium may reside in an ASIC. The ASIC mayreside in a user terminal. In the alternative, the processor and thestorage medium may reside as discrete components in a user terminal.

In one or more exemplary embodiments, the functions described may beimplemented in hardware, software, firmware, or any combination thereof.If implemented in software as a computer program product, the functionsmay be stored on or transmitted over as one or more instructions or codeon a non-transitory computer-readable medium. Computer-readable mediacan include both computer storage media and communication mediaincluding any medium that facilitates transfer of a computer programfrom one place to another. A storage media may be any available mediathat can be accessed by a computer. By way of example, and notlimitation, such non-transitory computer-readable media can compriseRAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic diskstorage or other magnetic storage devices, or any other medium that canbe used to carry or store desired program code in the form ofinstructions or data structures and that can be accessed by a computer.Also, any connection is properly termed a computer-readable medium. Forexample, if the software is transmitted from a web site, server, orother remote source using a coaxial cable, fiber optic cable, twistedpair, digital subscriber line (DSL), or wireless technologies such asinfrared, radio, and microwave, then the coaxial cable, fiber opticcable, twisted pair, DSL, or wireless technologies such as infrared,radio, and microwave are included in the definition of medium. Disk anddisc, as used herein, includes compact disc (CD), laser disc, opticaldisc, digital versatile disc (DVD), floppy disk and blu-ray disc wheredisks usually reproduce data magnetically, while discs reproduce dataoptically with lasers. Combinations of the above should also be includedwithin the scope of non-transitory computer-readable media.

The previous description of the disclosed embodiments is provided toenable any person skilled in the art to make or use the presentinvention. Various modifications to these embodiments will be readilyapparent to those skilled in the art, and the generic principles definedherein may be applied to other embodiments without departing from thespirit or scope of the invention. Thus, the present invention is notintended to be limited to the embodiments shown herein but is to beaccorded the widest scope consistent with the principles and novelfeatures disclosed herein.

What is claimed is:
 1. A method for performing context inference for amobile device comprising: receiving a data sample from at least a firstsensor; performing a first classification of the data sample from thefirst sensor; determining a confidence value for the firstclassification, wherein determining the confidence value comprises:detecting a change in classification from a prior data sampleclassification, of a prior data sample from the first sensor, to thefirst data sample classification, wherein the change in classificationmatches a change in classification predetermined to trigger a confidencefailure; activating a second sensor based on a determination theconfidence value of the first classification fails to meet a confidencethreshold; and performing a second classification of a second datasample from the activated second sensor jointly with the data from theat least first sensor.
 2. The method of claim 1, further comprising:determining a confidence value for the second classification; activatinga third sensor based on a determination that the confidence value failsto meet a confidence threshold; and performing a third classification ofa third data sample.
 3. The method of claim 1, wherein receiving datafrom at least a first sensor comprises: receiving data from two or moresensors.
 4. The method of claim 1, wherein the confidence value is basedon at least one of: a probability of correctly classifying the firstdata sample, a comparison of the first data sample to a data sample in ahistorical data set, and a difference between a nearest classificationand a next nearest classification.
 5. The method of claim 1, wherein thefirst sensor is part of a first sensor subsystem and the second sensoris part of a second sensor subsystem, and wherein activating the secondsensor further comprises: powering up the second sensor subsystem, andcommunicating between the first and second sensor subsystems.
 6. Amachine readable non-transitory storage medium containing executableprogram instructions which cause a mobile device to perform a method forperforming context inference for the mobile device, the methodcomprising: receiving a data sample from at least a first sensor;performing a first classification of the data sample from the firstsensor; determining a confidence value for the first classification,wherein determining the confidence value comprises: detecting a changein classification from a prior data sample classification, of a priordata sample from the first sensor, to the first data sampleclassification, wherein the change in classification matches a change inclassification predetermined to trigger a confidence failure; activatinga second sensor based on a determination the confidence value of thefirst classification fails to meet a confidence threshold; andperforming a second classification of a second data sample from theactivated second sensor jointly with the data from the at least firstsensor.
 7. The machine readable non-transitory storage medium of claim6, further comprising: determining a confidence value for the secondclassification; activating a third sensor based on a determination thatthe confidence value fails to meet a confidence threshold; andperforming a third classification of a third data sample.
 8. The machinereadable non-transitory storage medium of claim 6, wherein receivingdata from at least a first sensor comprises: receiving data from two ormore sensors.
 9. The machine readable non-transitory storage medium ofclaim 6, wherein the confidence value is based on at least one of: aprobability of the classification occurring, a comparison of the firstdata sample to a data sample in a historical data set, and a differencebetween a nearest classification and a next nearest classification. 10.The machine readable non-transitory storage medium of claim 6, whereinthe first sensor is part of a first sensor subsystem and the secondsensor is part of a second sensor subsystem, and wherein activating thesecond sensor further comprises: powering up the second sensorsubsystem, and communicating between the first and second sensorsubsystems.
 11. A data processing device comprising: a processor; astorage device coupled to the processor and configurable for storinginstructions, which, when executed by the processor cause the processorto: receive a data sample from at least a first sensor; perform a firstclassification of the data sample from the first sensor; determine aconfidence value for the first classification, wherein to determine theconfidence value comprises: detect a change in classification from aprior data sample classification, of a prior data sample from the firstsensor, to the first data sample classification, wherein the change inclassification matches a change in classification predetermined totrigger a confidence failure; activate a second sensor based on adetermination the confidence value of the first classification fails tomeet a confidence threshold; and perform a second classification of asecond data sample from the activated second sensor jointly with thedata from the at least first sensor.
 12. The data processing device ofclaim 11, further comprising instructions to: determine a confidencevalue for the second classification; activate a third sensor based on adetermination that the confidence value fails to meet a confidencethreshold; and perform a third classification of a third data sample.13. The data processing device of claim 11, wherein receiving data fromat least a first sensor further comprises instructions to: receive datafrom two or more sensors.
 14. The data processing device of claim 11,wherein the confidence value is based on at least one of: a probabilityof the classification occurring, a comparison of the first data sampleto a data sample in a historical data set, and a difference between anearest classification and a next nearest classification.
 15. The dataprocessing device of claim 11, wherein the first sensor is part of afirst sensor subsystem and the second sensor is part of a second sensorsubsystem, and wherein activating the second sensor further comprisesinstructions to: power up the second sensor subsystem, and communicatebetween the first and second sensor subsystems.
 16. An apparatus forperforming context inference for a mobile device comprising: means forreceiving data from at least a first sensor; means for performing afirst classification of a data sample from the first sensor; means fordetermining a confidence value for the first classification, whereindetermining the confidence value comprises: detecting a change inclassification from a prior data sample classification, of a prior datasample from the first sensor, to the first data sample classification,wherein the change in classification matches a change in classificationpredetermined to trigger a confidence failure; means for activating asecond sensor based on a determination the confidence value of the firstclassification fails to meet a confidence threshold; and means forperforming a second classification of a second data sample from theactivated second sensor jointly with the data from the at least firstsensor.
 17. The apparatus of claim 16, further comprising: means fordetermining a confidence value for the second classification; means foractivating a third sensor based on a determination that the confidencevalue fails to meet a confidence threshold; and means for performing athird classification of a third data sample.
 18. The apparatus of claim16, wherein receiving data from at least a first sensor comprises:receiving data from two or more sensors.
 19. The apparatus of claim 16,wherein the confidence value is based on at least one of: a probabilityof the classification occurring, a comparison of the first data sampleto a data sample in a historical data set, and a difference between anearest classification and a next nearest classification.
 20. Theapparatus of claim 16, wherein the first sensor is part of a firstsensor subsystem and the second sensor is part of a second sensorsubsystem, and wherein activating the second sensor further comprises:means for powering up the second sensor subsystem, and means forcommunicating between the first and second sensor subsystems.